Add autosolve actions for automated issue resolution by fantapop · Pull Request #14 · cockroachdb/actions

fantapop · 2026-03-25T00:25:39Z

Summary

Go implementation of composite actions for Claude-powered automated issue resolution:

autosolve/assess — Runs Claude in read-only mode to evaluate whether a task is suitable for automated resolution
autosolve/implement — Runs Claude to implement a solution, validates changes, runs AI security review, pushes to a fork, and creates a draft PR

Key features

Per-file batched AI security review with generated-file detection
Token usage tracking across phases with combined markdown summary
Retry logic with Claude session resumption
Skill file support for custom prompts

Testing

Tested end-to-end against cockroachlabs/ccloud-private-automation-testing.

Test plan

go test ./... passes
Precompiled binary check passes in CI

Copilot

Pull request overview

This PR introduces a new autosolve Go-based automation tool and two composite GitHub Actions (autosolve/assess and autosolve/implement) to assess issue suitability and implement fixes with Claude, including PR creation, security checks, and usage tracking.

Changes:

Add Go implementation for assessment/implementation orchestration, prompt assembly, git/gh wrappers, and security checks.
Add composite actions (autosolve/assess, autosolve/implement) plus CI updates to run Go tests and validate the precompiled binary.
Add prompt templates and unit tests for core functionality.

Reviewed changes

Copilot reviewed 28 out of 30 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
autosolve/internal/security/security_test.go	Adds unit tests for blocked-path and sensitive-file enforcement and `.gitignore` warnings.
autosolve/internal/security/security.go	Implements blocked path checks, sensitive filename/extension detection, and symlink-to-blocked-path detection.
autosolve/internal/prompt/templates/security-preamble.md	Adds system preamble intended to constrain the model’s behavior for safety.
autosolve/internal/prompt/templates/implementation-footer.md	Adds implementation-phase instruction footer and required success/fail marker.
autosolve/internal/prompt/templates/assessment-footer.md	Adds assessment-phase instruction footer and required proceed/skip marker.
autosolve/internal/prompt/prompt_test.go	Adds tests for prompt construction, skill file inclusion, and custom criteria.
autosolve/internal/prompt/prompt.go	Implements prompt assembly from templates + task inputs.
autosolve/internal/implement/implement_test.go	Adds tests for retry logic, output writing, and summary extraction.
autosolve/internal/implement/implement.go	Implements the implementation phase: retries, security checks, staging/commit/push, PR creation, and AI security review.
autosolve/internal/github/github.go	Adds a `gh`-CLI-backed GitHub client for comments/labels/PR creation.
autosolve/internal/git/git.go	Adds a git CLI abstraction and helper to list changed files.
autosolve/internal/config/config_test.go	Adds tests for config parsing/validation and blocked path parsing.
autosolve/internal/config/config.go	Adds config loading/validation from action inputs and auth validation.
autosolve/internal/claude/claude_test.go	Adds tests for extracting markers/session IDs and usage tracking.
autosolve/internal/claude/claude.go	Adds Claude CLI runner + result parsing + usage tracking persistence.
autosolve/internal/assess/assess_test.go	Adds tests for assessment flow and summary extraction.
autosolve/internal/assess/assess.go	Implements assessment phase invocation and outputs/summary writing.
autosolve/internal/action/action_test.go	Adds tests for GitHub Actions output and step summary helpers.
autosolve/internal/action/action.go	Adds helpers for outputs, summaries, and workflow annotations.
autosolve/implement/action.yml	Defines the composite action to run `autosolve implement` and expose outputs.
autosolve/go.mod	Introduces the autosolve Go module definition.
autosolve/cmd/autosolve/main.go	Adds CLI entrypoint for `assess` and `implement` commands.
autosolve/build.sh	Adds cross-compile script producing the committed Linux binary.
autosolve/assess/action.yml	Defines the composite action to run `autosolve assess` and expose outputs.
autosolve/Makefile	Adds build/test/clean targets for local development and CI.
autosolve/.gitignore	Ignores the local dev binary output.
CHANGELOG.md	Documents the addition of the autosolve actions.
.github/workflows/test.yml	Updates CI to run Go tests and ensure the precompiled binary is up to date.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fantapop · 2026-03-27T22:30:09Z

One thing I'm running into here is that build the action and committing it each time easily gets out of date and is annoying. I'm going to look into alternatives.

linhcrl

Left a couple of comments. Most of them are smaller/questions.

Also, here's some feedback I didn't know where to put:

In the PR description I see Precompiled Go binary (no Go toolchain needed at runtime) and one of the bottom checkboxes also mentions precompiled go binary. I'm assuming this just hasn't been updated right? I see that we actually recompile the binary every time this action is run
We should add some documentation in the README

Replace the boolean verbose_logging flag with a three-level log_level input (error/info/debug) that controls how much Claude CLI output streams to the GitHub Actions step log in real time. - error (default): silent, only errors and final status - info: pretty-printed result summary, permission denial warnings - debug: full stream including all tool calls, text, and results Permission denials are parsed from the result object and logged via ::warning:: annotations at info and debug levels. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

linhcrl · 2026-05-07T18:54:43Z

+| `context_vars`        | `""`                 | Comma-separated list of environment variable names to pass through to Claude for untrusted user input (e.g., issue titles/bodies)                     |
+| `assessment_criteria` | `""`                 | Custom criteria for the assessment. Uses default criteria if not provided.                                                                            |
+| `model`               | `claude-opus-4-6`    | Claude model ID                                                                                                                                       |
+| `blocked_paths`       | `.github/workflows/` | Comma-separated path prefixes that cannot be modified. `.github/` is always blocked.                                                                 |


if .github/ is always blocked, then setting .github/workflows/ as the default feels redundant. Should the default just be ""

Good point — `.github/` is always added by `requiredBlockedPaths`, so the default of `.github/workflows/` is redundant. Will update the default to empty. 🤖

I went through and replaced ./github/workflows/ with .github anywhere appropriate. There are still some references to files under ./github/workflows/ but the configuration is using ".github/" as the not allowed directory to catch those.

linhcrl · 2026-05-07T18:59:48Z

+| `system_prompt`      | `""`                             | Trusted instructions for Claude describing the task. Do not embed untrusted user input — use `context_vars`.    |
+| `skill`              | `""`                             | Path to a skill/prompt file relative to the repo root                                                           |
+| `context_vars`       | `""`                             | Comma-separated list of environment variable names to pass through to Claude for untrusted user input           |
+| `allowed_tools`      | `Read,Write,Edit,Grep,Glob,...`  | Claude `--allowedTools` string (defaults include git, go build/test/vet, and make)                              |


Should the full list of defaults be included here or at least link to a page where the full list can be viewed if too long

Will add the full list of always-blocked paths to the README so users know what's enforced regardless of their config. 🤖

Okay, this is there now

linhcrl · 2026-05-07T19:08:53Z

I think it would be helpful to somehow indicate which fields are required

Will mark required fields in the input tables. For both actions, `system_prompt` or `skill` (at least one) and `model` are required. 🤖

okay, now instead of a default for required fields it either says required or one required with an additional explanation

linhcrl · 2026-05-07T20:08:39Z

+- `autosolve/assess` action: evaluate tasks for automated resolution suitability
+  using Claude in read-only mode.
+- `autosolve/implement` action: autonomously implement solutions, validate
+  security, push to fork, and create PRs using Claude. Includes AI security
+  review, token usage tracking, per-file batched diff analysis, and structured
+  log levels (error/info/debug) with permission denial warnings.


These should be moved to the top of the section

Will fix the ordering — newer entries should be above older ones within each section. 🤖

this is fixed now

linhcrl · 2026-05-07T22:17:11Z

+	action.LogInfo(fmt.Sprintf("%s usage: input=%d output=%d cost=$%.4f",
+		section, result.Usage.InputTokens, result.Usage.OutputTokens, result.Usage.CostUSD))
+	if result.PermissionDenials > 0 && logLevel != "error" {
+		action.LogWarning(fmt.Sprintf("%s: %d tool call(s) were denied by permission policy",


Should we include the tools denied in the log? It might be helpful for troubleshooting

we're actually logging the denied tools elsewhere at info and debug level. I do think it's useful to see at error level as well so i'll just open that up and log it for all log levels.

by that I mean I'll keep logging the count here but at all log levels. The actual tools can be inspected elsewhere in info and debug level.

Check if the branch already exists on the fork remote before spending tokens on Claude. This avoids the frustrating case where implementation succeeds but PR creation fails because a previous run left the branch. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

The AI security review receives attacker-controlled content (staged diffs) in its prompt. Restrict its Bash access to git diff and git show commands to prevent prompt injection from escalating to arbitrary shell execution. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Add nil guard to LogResult to prevent panic when runner returns (nil, err). Return errors from usage and permission denial JSON parsing instead of silently swallowing them. Move tracker.Record in security review after the error check. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Flag symlinks resolving outside the repository root as a security violation, not just symlinks into blocked paths. Covers both absolute and relative symlink targets. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

PR creation is the only supported mode — there's no use case for running implement without creating a PR. This simplifies config validation and the README by making fork_owner, fork_repo, fork_push_token, and pr_create_token always required. Also updates blocked_paths default from .github/workflows/ to empty since .github/ is always blocked by requiredBlockedPaths. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

- assess tests now assert actual output values (PROCEED/SKIP) - Add duplicate Record test documenting accumulation behavior - Add case-sensitivity test for blocked paths - Consolidate autosolve changelog into single unreleased entry - Document that blocked_paths matching is case-sensitive Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Mark system_prompt and skill as "at least one required" in both action input tables. Change skill path description from "relative to the repo root" to "relative to working_directory". Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

…ions Label creation no longer fails the action if the token lacks permission (e.g. SSO enforcement). Adds a Token permissions section to the README documenting required scopes and SAML/SSO authorization. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

fantapop · 2026-05-08T21:20:50Z

Left a couple of comments. Most of them are smaller/questions.

Also, here's some feedback I didn't know where to put:

In the PR description I see Precompiled Go binary (no Go toolchain needed at runtime) and one of the bottom checkboxes also mentions precompiled go binary. I'm assuming this just hasn't been updated right? I see that we actually recompile the binary every time this action is run

We should add some documentation in the README

I think 1 is cleaned up now. I don't see any of that. 2. was added as you've reviewed later.

The skill file path is now resolved relative to GITHUB_WORKSPACE instead of the working directory, so workflow authors always specify paths from the workspace root regardless of working_directory. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Rename output files from generic names to assess-claude-output.json, implement-claude-output.json, and security-review-claude-output.json. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Define a sentinel error for empty Claude results so tests can assert with errors.Is instead of string matching. Remove unnecessary BlockedPaths from assess test configs. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Add direct tests for aiSecurityReview covering nothing-staged, pass, and fail-with-reset paths. Fix mock to only write metadata files on IMPLEMENTATION_RESULT, preventing leaked .autosolve-* artifacts. Use a cleanup helper for robustness. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Break the monolithic pushAndPR into setupForkRemote, stageChanges, validateChanges, commitAndPush, and createPR. Require an explicit PR title or commit subject instead of falling back to the system prompt. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Change test blocked paths from .github/workflows/ to .github/ to match the production default. Remove unnecessary BlockedPaths from implement test configs. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Move the long allowed_tools default out of the table into a referenced section below, matching the pr_footer pattern. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Move default assessment criteria from a Go constant into an embedded template file so the README can link directly to it. Note that assessment_criteria is trusted input. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

These inputs are unused and add unnecessary complexity. PR titles are derived from the commit subject line, and PR bodies are always written by Claude to .autosolve-pr-body. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Add a Classic column to the token permissions table showing that both tokens need the `repo` scope. Simplify the label permission note and drop the "pre-create labels" workaround. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Check os.Setenv return value in ensureTmpDir, explicitly discard os.Unsetenv errors with _ =, and replace hand-rolled contains helper with slices.Contains from the standard library. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

fantapop force-pushed the pr/autosolve-go branch from 324a6db to 0655ce2 Compare March 25, 2026 00:29

fantapop force-pushed the pr/shell-framework branch from 9134765 to 817680c Compare March 25, 2026 00:38

fantapop force-pushed the pr/autosolve-go branch from 0655ce2 to 6f1121d Compare March 25, 2026 00:38

fantapop force-pushed the pr/shell-framework branch from 817680c to 0a678c6 Compare March 25, 2026 01:00

fantapop force-pushed the pr/autosolve-go branch from 6f1121d to 5c7a16f Compare March 25, 2026 01:03

fantapop changed the base branch from pr/shell-framework to main March 25, 2026 01:03

This was referenced Mar 25, 2026

Generic autosolve github workflow for automated issue resolution #5

Closed

Add Go rewrite of autosolve actions #8

Closed

fantapop requested a review from Copilot March 25, 2026 01:24

Copilot started reviewing on behalf of fantapop March 25, 2026 01:25 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

fantapop force-pushed the pr/autosolve-go branch from 5c7a16f to a9a9010 Compare March 25, 2026 01:31

fantapop requested a review from linhcrl March 25, 2026 01:35

fantapop force-pushed the pr/autosolve-go branch 2 times, most recently from 1abbbb0 to 6fd24ba Compare March 25, 2026 06:52

fantapop commented Mar 25, 2026

View reviewed changes

Comment thread .github/workflows/test.yml Outdated

fantapop force-pushed the pr/autosolve-go branch 6 times, most recently from f818651 to 6bc6bc5 Compare March 27, 2026 22:21

fantapop force-pushed the pr/autosolve-go branch 2 times, most recently from d06e466 to f2ef7a1 Compare March 27, 2026 22:50

linhcrl reviewed Apr 2, 2026

View reviewed changes

fantapop force-pushed the pr/autosolve-go branch 4 times, most recently from 981e842 to a89fb9b Compare April 8, 2026 06:45

fantapop force-pushed the pr/autosolve-go branch from 8c632cc to 508b7ea Compare May 7, 2026 01:09

fantapop force-pushed the pr/autosolve-go branch from 508b7ea to 9adf4f9 Compare May 7, 2026 18:49

linhcrl reviewed May 7, 2026

View reviewed changes

fantapop force-pushed the pr/autosolve-go branch from 42dd416 to 51c0306 Compare May 7, 2026 23:15

fantapop and others added 8 commits May 7, 2026 16:52

autosolve: Update Go version to 1.26

b793a01

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

autosolve: Log roachdev version when using wrapper

4084a28

Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

fantapop force-pushed the pr/autosolve-go branch from 1f68519 to 5a2aa7a Compare May 8, 2026 18:46

fantapop force-pushed the pr/autosolve-go branch from 41f331b to c863d36 Compare May 8, 2026 21:10

fantapop and others added 5 commits May 8, 2026 14:48

autosolve: Use descriptive names for Claude output files

7f6b59e

Rename output files from generic names to assess-claude-output.json, implement-claude-output.json, and security-review-claude-output.json. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

fantapop force-pushed the pr/autosolve-go branch from 771caf1 to a42f94b Compare May 8, 2026 23:24

fantapop and others added 6 commits May 8, 2026 16:30

autosolve: Update tests to use .github/ blocked path

ca4d75e

Change test blocked paths from .github/workflows/ to .github/ to match the production default. Remove unnecessary BlockedPaths from implement test configs. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

autosolve: Document full allowed_tools default in README

8174636

Move the long allowed_tools default out of the table into a referenced section below, matching the pr_footer pattern. Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>

Conversation

fantapop commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key features

Testing

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fantapop commented Mar 27, 2026

Uh oh!

linhcrl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fantapop commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fantapop commented Mar 25, 2026 •

edited

Loading

linhcrl left a comment •

edited

Loading